feat(cli): okdev restart and short-name pod addressing#158
Merged
Conversation
Field feedback asked for a "post-up hook" — which spec.lifecycle postCreate/postSync already are (their done-markers are pod annotations, so recreated pods re-run them automatically). The gap was discoverability: neither the skill guidance nor the reconcile troubleshooting mentioned them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two recovery-experience fixes for multi-pod training sessions: okdev restart [session]: one-command recovery when a container died and the restart policy will not bring it back — delete the workload, wait for termination, reset local per-session sync state, and run the full up flow with the same config. PVCs are untouched; lifecycle hooks re-run on the fresh pods. Prompts unless --yes. Short-name pod addressing: --pod on exec/cp/jobs and target set --pod now accept the short names shown by `okdev status` (master-0, worker-1) and any unique "-<name>" suffix, in addition to full pod names — so scripts survive the pod-name hash changing on every recreation. Unknown or ambiguous names error with the available aliases instead of the old silent-drop behavior; --exclude accepts the same aliases but tolerates unknown names (an excluded pod may no longer exist). --group already accepted short names; this makes the rest of the surface consistent. Docs and skill updated: restart reference, alias forms as the preferred addressing style, and the lifecycle-hook discoverability note (field feedback asked for a "post-up hook" that postCreate/postSync already provide — the gap was documentation). Kind e2e: smoke gains a restart scenario (pod UID must change, exec works after); the PyTorchJob workflow verifies master-0/worker-1 addressing end-to-end plus the unknown-name error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes out feedback item 4 (dead container → costly session rebuild; pod hash names invalidate every script on recreation).
okdev restart [session] [--yes] [--wait-timeout]One-command recovery when a container died and the restart policy won't bring it back: delete the workload → wait for termination → reset local per-session sync state → full
upflow with the same config. PVCs untouched; lifecycle hooks (postCreate/postSync) re-run on the fresh pods; sync re-bootstraps. Prompts unless--yes(refuses non-interactive without it).Short-name pod addressing
--podonexec/cp/jobs ...andtarget set --podnow accept, in order: the full pod name, the short name shown byokdev status(master-0,worker-1), or any unique-<name>suffix — so scripts survive the pod-name hash changing on every recreation.--excludeaccepts the same aliases but tolerates unknown names (an excluded pod may legitimately no longer exist).--groupalready supported short names; this makes the whole surface consistent.Docs/skill
Command reference (restart + alias forms), skill guidance (recovery path, aliases as the preferred addressing style per the original feedback), and the lifecycle-hook discoverability note — the requested "post-up hook" is what
spec.lifecycle.postCreate/postSyncalready do (their done-markers are pod annotations, so recreated pods re-run them); the gap was documentation.Testing
--pod master-0/--pod worker-1reach the right pods end-to-end and that an unknown short name errors with guidance.gofmt -lclean;go test -race ./...passes.🤖 Generated with Claude Code